Extraction system, walking training system, extraction method, and extraction program

ABSTRACT

The present disclosure provides an extraction system capable of accurately extracting a region of an image where a leg part of a trainee is included. an extraction system including: an image acquisition unit that acquires an image obtained by capturing the trainee; an extraction unit that sets regions of the image on both left and right sides of the belt of the treadmill to regions of the image where the leg part of the trainee is not present and thereby distinguishes the set regions from the region of the image where the leg part of the trainee is included, and extracts the region of the image where the leg part of the trainee is included; and a mask processing unit that applies a mask to each of the regions of the image where the leg part of the trainee is not present.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese patent application No. 2020-200982, filed on Dec. 3, 2020, thedisclosure of which is incorporated herein in its entirety by reference.

BACKGROUND

The present disclosure relates to an extraction system, a walkingtraining system, an extraction method, and an extraction program, and,for example, relates to an extraction system, a walking training system,an extraction method, and an extraction program that extract a region ofan image where a leg part of a trainee is included, the image beingobtained by capturing the trainee who performs walking training on abelt of a treadmill of a walking training apparatus.

In recent years, a walking training apparatus has sometimes been usedfor rehabilitation of a hemiplegic patient or the like. As disclosed inJapanese Patent No. 6597275, a common walking training apparatus has aconfiguration in which a belt of a treadmill exposed from an uppersurface of a main body part thereof is rotated, and a trainee such as ahemiplegic patient performs walking training on the belt.

SUMMARY

The applicant has however found the following problem. When a traineeperforms walking training on the belt, skeletal coordinates of thetrainee may be estimated in order to check whether or not the traineehas been separated from the belt, and at this time, an image is acquiredby capturing the trainee during the walking training.

When the trainee performs walking training on the belt as describedabove, an assistant supports the trainee while the feet of the assistantare placed on the regions of the main body part of the treadmill on bothright and left sides of the belt. Consequently, the feet of theassistant may be present in the image in addition to the trainee, andthus it may be difficult to accurately estimate the skeletal coordinatesof the trainee. Therefore, there has been a demand for a techniquecapable of accurately extracting a region of an image where a leg partof a trainee is included.

The present disclosure has been made in view of the above-describedproblem and provides an extraction system, a walking training system, anextraction method, and an extraction program that are capable ofaccurately extracting a region of an image where a leg part of a traineeis included.

A first exemplary aspect is an extraction system configured to extract aregion of an image where a leg part of a trainee is included, the imagebeing obtained by capturing the trainee who performs walking training ona belt of a treadmill of a walking training apparatus, the extractionsystem including:

an image acquisition unit configured to acquire an image obtained bycapturing the trainee from a front side or a rear side of the trainee;

an extraction unit configured to set regions of the image on both leftand right sides of the belt of the treadmill to regions of the imagewhere the leg part of the trainee is not present and thereby distinguishthe set regions from the region of the image where the leg part of thetrainee is included, and extract the region of the image where the legpart of the trainee is included; and

a mask processing unit configured to apply a mask to each of the regionsof the image where the leg part of the trainee is not present.

The above-described extraction system further includes a skeletonacquisition unit configured to acquire skeletal coordinates of the legpart of the trainee in real time,

in which the extraction unit changes the regions of the image where theleg part of the trainee is not present in real time so that the regionsof the image where the leg part of the trainee is not present areseparated from the leg part of the trainee by a predetermined distancebased on the skeletal coordinates of the leg part of the trainee.

In the above-described extraction system, the extraction unit changesthe regions of the image where the leg part of the trainee is notpresent in real time so that the regions of the image where the leg partof the trainee is not present are separated from a straight line by thepredetermined distance, the straight line connecting a position of a hipjoint of the trainee to a position of a knee joint of the trainee.

In the above-described extraction system, the mask processing unit setsa color of the mask based on clothes or shoes worn by the trainee.

Another exemplary aspect is a walking training system including:

the above-described extraction system; and a walking training apparatusincluding a treadmill for the trainee to perform walking training, inwhich

the walking training apparatus further includes:

a skeleton estimation unit configured to estimate the skeletalcoordinates of the leg part of the trainee based on the image to whichthe mask has been applied; and

an abnormality determination unit configured to determine whether or notan abnormality of the trainee has occurred based on the skeletalcoordinates of the leg part of the trainee.

Another exemplary aspect is an extraction method for extracting a regionof an image where a leg part of a trainee is included, the image beingobtained by capturing the trainee who performs walking training on abelt of a treadmill of a walking training apparatus, the extractionmethod including:

acquiring an image obtained by capturing the trainee from a front sideor a rear side of the trainee;

setting regions of the image on both left and right sides of the belt ofthe treadmill to regions of the image where the leg part of the traineeis not present, thereby distinguishing the set regions from the regionof the image where the leg part of the trainee is included, andextracting the region of the image where the leg part of the trainee isincluded; and

applying a mask to each of the regions of the image where the leg partof the trainee is not present.

The above-described extraction method further including estimatingskeletal coordinates of the leg part of the trainee in real time,

in which the regions of the image where the leg part of the trainee isnot present are changed in real time so that the regions of the imagewhere the leg part of the trainee is not present are separated from theleg part of the trainee by a predetermined distance based on theskeletal coordinates of the leg part of the trainee.

In the above-described extraction method, the regions of the image wherethe leg part of the trainee is not present are changed in real time sothat the regions of the image where the leg part of the trainee is notpresent are separated from a straight line by the predetermineddistance, the straight line connecting a position of a hip joint of thetrainee to a position of a knee joint of the trainee.

In the above-described extraction method, a color of the mask is setbased on clothes or shoes worn by the trainee.

Another exemplary aspect is an extraction program for extracting aregion of an image where a leg part of a trainee is included, the imagebeing obtained by capturing the trainee who performs walking training ona belt of a treadmill of a walking training apparatus, the extractionprogram causing a computer to:

acquire an image obtained by capturing the trainee from a front side ora rear side of the trainee;

set regions of the image on both left and right sides of the belt of thetreadmill to regions of the image where the leg part of the trainee isnot present and thereby distinguish the set regions from the region ofthe image where the leg part of the trainee is included, and extract theregion of the image where the leg part of the trainee is included; and

apply a mask to each of the regions of the image where the leg part ofthe trainee is not present.

According to the present disclosure, it is possible to provide anextraction system, a walking training system, an extraction method, andan extraction program that are capable of accurately extracting a regionof an image where a leg part of a trainee is included.

The above and other objects, features and advantages of the presentdisclosure will become more fully understood from the detaileddescription given hereinbelow and the accompanying drawings which aregiven by way of illustration only, and thus are not to be considered aslimiting the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing a state in which a trainee performs walkingtraining using a walking training system;

FIG. 2 is a block diagram of a control system of the walking trainingsystem according to an embodiment;

FIG. 3 is a diagram showing an image obtained by capturing the traineeduring walking training;

FIG. 4 is a flowchart showing a flow of the trainee performing walkingtraining using the walking training system according to the embodiment;

FIG. 5 is a diagram showing a region of a two-dimensional image where aleg part of the trainee is not present and a region of thetwo-dimensional image where the leg part of the trainee is included;

FIG. 6 is a diagram for explaining a method for setting the region ofthe two-dimensional image where the leg part of the trainee is notpresent;

FIG. 7 is a diagram showing in a stepwise manner a state in which theregion where the leg part of the trainee is not present is deformed inaccordance with a position of the trainee; and

FIG. 8 is a diagram showing an example of a hardware configurationincluded in the extraction system.

DESCRIPTION OF EMBODIMENTS

Specific embodiments to which the present disclosure is applied will bedescribed hereinafter in detail with reference to the drawings. However,the present disclosure is not limited to the embodiments shown below.Further, for the clarification of the description, the followingdescriptions and the drawings are simplified as appropriate.

First, a configuration of a walking training system using an extractionsystem according to this embodiment will be described. FIG. 1 is adiagram showing a state in which a trainee performs walking trainingusing the walking training system. FIG. 2 is a block diagram of acontrol system of the walking training system according to thisembodiment. In the following description, for the clarification of thedescription, a side of the walking training apparatus facing the face ofthe trainee when the trainee performs walking training will be describedas being the front side of the walking training apparatus.

As shown in FIGS. 1 and 2, a walking training system 1 includes awalking assistance apparatus 2, a walking training apparatus 3, acontrol apparatus 4, and an extraction system 5. The walking assistanceapparatus 2 is an assistance apparatus attached to a diseased leg of atrainee T for assisting walking training.

For example, the walking assistance apparatus 2 includes an upper leglink 11 attached to an upper leg part of the trainee T, a lower leg link12 that is rotatably connected to the upper leg link 11 and attached toa lower leg part of the trainee T, a foot link 13 that is rotatablyconnected to the lower leg link 12 and attached to a foot, and a motor14 that drives the upper leg link 11 and the lower leg link 12 so as torotate them relative to each other.

The walking training apparatus 3 includes a main body part 21, a beltpart 22, a frame part 23, a handrail part 24, a first leg load reliefpart 25, a second leg load relief part 26, a suspension load relief part27, a display 28, and an image capturing apparatus 29, and is formed asa tread mill.

The main body part 21 basically has a box shape, and the belt part 22 ishoused inside the main body part 21. An opening part 21 a from which anupper surface of the belt part 22 is exposed is formed on an uppersurface of the main body part 21. The regions of the main body part 21on both the left and right sides of the opening part 21 a can functionas steps on which the feet of an assistant are placed.

The belt part 22 is a training part for the trainee T to perform walkingtraining. The belt part 22 includes a belt 22 a and a motor 22 b. Thebelt 22 a is formed of an endless belt, and is supported by the mainbody part 21 so that an upper surface of the belt 22 a can be rotated soas to move toward the rear side of the walking training apparatus 3. Themotor 22 b drives the belt 22 a so as to rotate it.

The frame part 23 is a frame of the walking training apparatus 3. Theframe part 23 includes a first arch part 23 a, a second arch part 23 b,a first beam part 23 c, and a second beam part 23 d. The first arch part23 a is substantially arched when viewed from the left-right directionof the walking training apparatus 3, and is provided in a region of themain body part 21 on the right side of the belt part 22. The second archpart 23 b is substantially arched when viewed from the left-rightdirection of the walking training apparatus 3, and is provided in aregion of the main body part 21 on the left side of the belt part 22.

The first beam part 23 c is extended in the left-right direction of thewalking training apparatus 3 and connects a front corner part of thefirst arch part 23 a to a front corner part of the second arch part 23b. The second beam part 23 d is extended in the left-right direction ofthe walking training apparatus 3 and connects a rear corner part of thefirst arch part 23 a to a rear corner part of the second arch part 23 b.

The handrail part 24 is grasped by the trainee T riding on the belt part22. The handrail part 24 is extended in the front-rear direction of thewalking training apparatus 3 and is provided in the walking trainingapparatus 3, for example, on the side of the diseased leg to which thewalking assistance apparatus 2 is attached. The handrail part 24 isprovided, for example, in a region of the main body part 21 on the rightside of the belt part 22. However, a pair of the handrail parts 24 maybe provided in the regions of the main body part 21 on both the left andright sides, respectively, of the belt part 22.

The first leg load relief part 25 relieves, when the trainee T swingsthe diseased leg, the load on the diseased leg. The first leg loadrelief part 25 includes a motor 25 a and a wire 25 b. The motor 25 a isprovided in the first beam part 23 c. The motor 25 a drives the wire 25b so as to wind and feed it. A tip part of the wire 25 b is connected toany point on the walking assistance apparatus 2.

The second leg load relief part 26 relieves, when the trainee T pullsback the diseased leg, the load on the diseased leg. The second leg loadrelief part 26 includes a motor 26 a and a wire 26 b. The motor 26 a isprovided in the second beam part 23 d. The motor 26 a drives the wire 26b so as to wind and feed it. A tip part of the wire 26 b is connected toany point on the walking assistance apparatus 2.

The suspension load relief part 27 relieves the weight of the trainee T.The suspension load relief part 27 includes a harness 27 a, a connectionjig 27 b, and a wire 27 c. The harness 27 a is worn by the trainee T.The connection jig 27 b is provided in the frame part 23 between thefirst beam part 23 c and the second beam part 23 d of the frame part 23so that it connects the first arch part 23 a to the second arch part 23b.

The wire 27 c is suspended from the connection jig 27 b, and a tip partof the wire 27 c is connected to the harness 27 a. By thisconfiguration, when the trainee T wears the harness 27 a, the weight ofthe trainee T is supported by the frame part 23 via the wire 27 c andthe connection jig 27 b.

The display 28 is provided on the front side of the frame part 23 sothat a display part 28 a of the display 28 faces the rear side of thewalking training apparatus 3. The display part 28 a of the display 28displays, for example, a training instruction to the trainee T.

The image capturing apparatus 29 includes, for example, a first camera29 a and a second camera 29 b. The first camera 29 a is, for example, anRGB camera, and captures a two-dimensional image of the trainee T who isperforming walking training in real time. The second camera 29 b is, forexample, an infrared camera, and captures a depth image of the trainee Twho is performing walking training in real time.

These first and second cameras 29 a and 29 b are provided, for example,on the frame part 23 so that the trainee T who is performing walkingtraining can be captured from the front side. The image capturingapparatus 29 is disposed, for example, near the display 28. However, theimage capturing apparatus 29 only needs to be configured and disposed sothat it can capture a two-dimensional image and a depth image of thetrainee T, and for example, the image capturing apparatus 29 may bedisposed so that it can capture the trainee T from behind. In short, theimage capturing apparatus 29 only needs to be disposed so that it cancapture the trainee T from a side opposite to the side of the trainee Twhere the assistant is present.

The control apparatus 4 includes a control unit 31, a skeletonestimation unit 32, and an abnormality determination unit 33, and isconnected to, for example, the walking assistance apparatus 2 and thewalking training apparatus 3 via a network. However, the controlapparatus 4 may be incorporated into the walking training apparatus 3.

The control unit 31 controls the display 28 so as to display a traininginstruction or the like on the display part 28 a of the display 28, andalso controls the motor 22 b so as to rotate the belt 22 a, for example,based on a training content set in accordance with a training level ofthe trainee T.

At this time, the control unit 31 controls the motor 14 of the walkingassistance apparatus 2 so as to assist movement of the diseased leg ofthe trainee T based on the training content. Further, the control unit31 controls the motor 25 a of the first leg load relief part 25 and themotor 26 a of the second leg load relief part 26 based on the trainingcontent.

That is, when the trainee T swings the diseased leg, the control unit 31controls the motor 25 a of the first leg load relief part 25 to wind thewire 25 b, and also controls the motor 26 a of the second leg loadrelief part 26 to feed the wire 26 b, thereby relieving the load on thediseased leg when the trainee T swings it.

On the other hand, when the trainee T pulls back the diseased leg, thecontrol unit 31 controls the motor 25 a of the first leg load reliefpart 25 to feed the wire 25 b, and also controls the motor 26 a of thesecond leg load relief part 26 to wind the wire 26 b, thereby relievingthe load on the diseased leg when the trainee T pulls it back.

The skeleton estimation unit 32 estimates skeletal coordinates of thetrainee T based on the two-dimensional image and the depth imagecaptured by the image capturing apparatus 29. As the skeletalcoordinates of the trainee T, for example, at least two-dimensionalcoordinates and a depth of the hip joint, two-dimensional coordinatesand a depth of the knee joint, and two-dimensional coordinates and adepth of the ankle joint are estimated.

At this time, as a method for estimating skeletal coordinates of thetrainee T, a general method, for example, a vision pose, can be used.Here, the two-dimensional coordinates are two-dimensional coordinates ofwhich the origin is a predetermined position in the two-dimensionalimage. Further, the depth is a depth of a position corresponding to thetwo-dimensional coordinates in the depth image or an average value ofthe depths in the vicinity of this position.

The abnormality determination unit 33 determines whether or not anabnormality has occurred in the walking of the trainee T based on theskeletal coordinates of the trainee T. The abnormality determinationunit 33 determines, for example, whether or not the trainee T is aboutto be separated from the belt 22 a in the left-right or rear directionthereof.

When the abnormality determination unit 33 determines that the trainee Tis about to be separated from the belt 22 a in the left-right or reardirection thereof, it determines that the walking of the trainee T isabnormal. On the other hand, when the abnormality determination unit 33determines that the trainee T is not about to be separated from the belt22 a in the left-right or rear direction thereof, it determines that thewalking of the trainee T is normal.

When the trainee T performs walking training using the above-describedwalking training system 1, an assistant supports the trainee T while thefeet of the assistant are placed on the regions of the main body part 21of the walking training apparatus 3 on both the right and left side ofthe belt 22 a.

FIG. 3 is a diagram showing an image obtained by capturing the traineeduring walking training. As described above, the skeleton estimationunit 32 estimates the skeletal coordinates of the trainee T based on thetwo-dimensional image and the depth image captured by the imagecapturing apparatus 29. However, as shown in FIG. 3, the feet of anassistant A may be present in the two-dimensional image in addition tothe trainee T, and thus it may be difficult to accurately acquireskeletal information of the trainee T.

Therefore, the walking training system 1 according to this embodimentincludes the extraction system 5. The extraction system 5 isincorporated into, for example, the control apparatus 4, and includes animage acquisition unit 41, a skeleton acquisition unit 42, an extractionunit 43, and a mask processing unit 44. Note that although theextraction system 5 is incorporated into the control apparatus 4 in thisembodiment, a different configuration may be employed.

The image acquisition unit 41 acquires the two-dimensional imagecaptured by the first camera 29 a. At this time, the image acquisitionunit 41 may acquire the two-dimensional image in real time. However, theimage acquisition unit 41 can be composed of the first camera 29 a, orcan be composed of another camera capable of capturing a two-dimensionalimage of the trainee T. Further, the image acquisition unit 41 mayacquire a two-dimensional image of the trainee T captured by anothercamera.

The skeleton acquisition unit 42 acquires the skeletal coordinates ofthe trainee T. The skeleton acquisition unit 42 acquires, for example,the skeletal coordinates of the trainee T estimated last time from theskeleton acquisition unit 42. However, the skeleton acquisition unit 42can be composed of the skeleton estimation unit 32, or can be composedof another skeleton estimation unit. Further, the skeleton acquisitionunit 42 may acquire the skeletal coordinates of the trainee T estimatedlast time by another skeleton estimation unit.

The extraction unit 43 sets a region of the two-dimensional image wherethe leg part of the trainee T is not present and thereby distinguishesthe region where the leg part of the trainee T is not present from theregion where the leg part of the trainee T is included, and extracts theregion where the leg part of the trainee T is included. The maskprocessing unit 44 applies a mask to the region of the two-dimensionalimage where the leg part of the trainee T is not present.

Next, a flow in which the trainee T performs walking training using thewalking training system 1 according to this embodiment will bedescribed. FIG. 4 is a flowchart showing a flow in which the traineeperforms walking training using the walking training system according tothis embodiment. FIG. 5 is a diagram showing a region of atwo-dimensional image where a leg part of the trainee is not present anda region of the two-dimensional image where the leg part of the traineeis included. FIG. 6 is a diagram for explaining a method for setting theregion of the two-dimensional image where the leg part of the trainee isnot present. FIG. 7 is a diagram showing in a stepwise manner a state inwhich the region where the leg part of the trainee is not present isdeformed in accordance with a position of the trainee. The stages ofdeformation of this region when the trainee moves on the belt of thewalking training apparatus from the center to the left thereof is shownin FIG. 7 in the order from left to right. Note that an assistant isomitted in FIG. 7.

Here, it is assumed that the trainee T performs walking training on thebelt 22 a, the image capturing apparatus 29 acquires an image of thestate of the trainee T, the skeleton estimation unit 32 estimatesskeletal coordinates of the trainee T based on the acquired image, andthe abnormality determination unit 33 determines whether or not anabnormality of the trainee T has occurred based on the estimatedskeletal coordinates of the trainee T.

In this state, first, the image acquisition unit 41 of the extractionsystem 5 acquires the two-dimensional image captured by the first camera29 a this time, and outputs information indicating the two-dimensionalimage captured this time to the extraction unit 43 and the maskprocessing unit 44 (S1). Next, the skeleton acquisition unit 42 of theextraction system 5 determines whether or not the skeletal coordinatesof the trainee T estimated last time have been successfully acquired(S2).

If the skeleton acquisition unit 42 of the extraction system 5determines that the skeletal coordinates of the trainee T estimated lasttime have been successfully acquired (YES in S2), it outputs informationindicating that a result of the determination is that the skeletalcoordinates of the trainee T have been successfully acquired to theextraction unit 43. If the extraction unit 43 of the extraction system 5receives the information indicating that the result of the determinationis that the skeletal coordinates of the trainee T have been successfullyacquired, it first sets, as shown in FIG. 5, the regions of thetwo-dimensional image on both the right and left sides of the belt 22 aas first regions R1 where the leg part of the trainee T is not present.

When the trainee T performs walking training as described above, forexample, the assistant A supports the trainee T from the rear side ofthe trainee T while the feet of the assistant A are placed on theregions of the main body part 21 on both the right and left sides of thebelt 22 a, and the trainee T performs the walking training on the belt22 a.

Therefore, the extraction unit 43 sets regions of the two-dimensionalimage on both the right and left sides of the belt 22 a as the firstregions R1. In this embodiment, as shown in FIG. 5, regions of thetwo-dimensional image on both the right and left sides of the belt 22 aare set to the first regions R1, which regions have heights near that ofthe handrail part 24 (in other words, heights the same as that of thewaist of the trainee T). At this time, each of the first regions R1 mayhave a shape obtained by cutting out a region where the trainee T isassumed to grasp the handrail part 24 in some embodiments.

For example, the first region R1 on the right side is a regionsurrounded by a first straight line L1 extending along a right end edgeof the belt 22 a, a second straight line L2 connecting a front end partof the first straight line L1 to a right corner part of thetwo-dimensional image, a third straight line L3 extending up to a heightthe same as the height of the handrail part 24 on the right side of thetwo-dimensional image, a notched line L4, a fourth straight line L5connecting an upper end part of the third straight line L3 to an upperend part of a vertical line L4 a in the notched line L4, and a fifthstraight line L6 connecting a rear end part of the first straight lineL1 to an end part of a horizontal line L4 b in the notched line L4.

On the other hand, since the first region R1 on the left side has ashape of line symmetrical with the first region R1 on the right sidewith respect to an axis extending in the front-rear direction andpassing through the center of the belt 22 a in the left-right directionas a symmetrical axis, a detailed description thereof will be omitted.Note that regions of the two-dimensional image where the belt 22 a, thehandrail part 24, and the like are present can be recognized byperforming image processing on the two-dimensional image. However, thefirst regions R1 having the above-described shapes may be set by aninput from the outside.

Here, since a second region R2 where the leg part of the trainee T isincluded changes in accordance with the walking state of the trainee Tduring the walking training, the extraction unit 43 of the extractionsystem 5 acquires information indicating the skeletal coordinates of thetrainee T estimated last time from the skeleton acquisition unit 42 andchange, based on the skeletal coordinates of the trainee T estimatedlast time, the first region R1 so that the first region R1 is separatedfrom the leg part of the trainee T by a predetermined distance in thetwo-dimensional image acquired this time.

For example, as shown in FIG. 6, the extraction unit 43 of theextraction system 5 can change the first region R1 so that the firstregion R1 is separated from a straight line L7 connecting thetwo-dimensional coordinates of a hip joint H of the trainee T to thetwo-dimensional coordinates of a knee joint K of the trainee T estimatedlast time by a predetermined distance D.

That is, as shown in FIG. 6, the extraction unit 43 of the extractionsystem 5 extends the first straight line L1 and the horizontal line L4 bof the notched line L4 of the first region R1 and shortens the fifthstraight line L6 of the first region R1, and shortens the first straightline L1 and the horizontal line L4 b of the notched line L4 of the firstregion R1 and extends the fifth straight line L6 of the first region R1so that the fifth straight line L6 of the first region R1 is positionedsubstantially in parallel with the straight line L7 and separated fromthe straight line L7 by the distance D, thereby changing the firstregion R1.

Specifically, as shown in the order from left to right in FIG. 7, whenthe trainee T moves on the belt 22 a from the right side to the leftside thereof, the extraction unit 43 of the extraction system 5 deformsthe first region R1 by shortening the first straight line L1 and thehorizontal line L4 b of the notched line L4 of the first region R1 andextending the fifth straight line L6 of the first region R1 whilemaintaining the state in which the fifth straight line L6 of the firstregion R1 is positioned substantially parallel to the straight line L7and separated from the straight line L7 by the distance D so that thefirst region R1 on one of the left side or the right side (in FIG. 7,the left side) to which the trainee T moves does not interfere with thetrainee T.

Meanwhile, the extraction unit 43 of the extraction system 5 deforms thefirst region R1 by extending the first straight line L1 and thehorizontal line L4 b of the notched line L4 of the first region R1 andshortening the fifth straight line L6 of the first region R1 whilemaintaining the state in which the fifth straight line L6 of the firstregion R1 on the other side (in FIG. 7, the right side) to which thetrainee T moves is positioned substantially in parallel with thestraight line L7 and separated from the straight line L7 by the distanceD.

By doing so, it is possible to accurately set the first region R1 bychanging it in real time in accordance with the walking state of thetrainee T. In other words, the extraction unit 43 can accurately extractthe second region R2 by changing it in real time. The extraction unit 43outputs information (e.g., two-dimensional coordinate information ofpositions where the lines of the first region R1 intersect each other)indicating the two-dimensional coordinates of the first region R1 to themask processing unit 44.

Next, the mask processing unit 44 of the extraction system 5 applies amask to the first region R1 in the two-dimensional image acquired thistime based on the two-dimensional image acquired this time and thetwo-dimensional coordinates of the first region R1 (S3). By doing so, itis possible to hide the first region R1 where the foot of the assistantA is present by the mask. The mask processing unit 44 outputsinformation indicating the two-dimensional image to which the mask hasbeen applied to the skeleton estimation unit 32.

Here, a color of the mask applied to the two-dimensional image can beappropriately set depending on colors and materials of clothes and shoesworn by the trainee T and the assistant A. However, the color of themask is, for example, pure white, which is not used as the colors of theclothes and the shoes, in order to prevent the first region R1 frombeing erroneously recognized as the second region R2. However, the maskprocessing unit 44 of the extraction system 5 may recognize the colorsof the clothes and the shoes worn by the trainee T and the assistant Abased on the two-dimensional image, and set the color of the mask to acolor different from the colors of the clothes and the shoes worn by thetrainee T and the assistant A. Further, the color of the mask may be setby an input from the outside.

Next, the skeleton estimation unit 32 estimates skeletal coordinates ofthe trainee T based on the two-dimensional image to which the mask hasbeen applied and the depth image acquired this time (S4). At this time,the mask has been applied to the first regions R1 including the regionson both the right and left sides of the belt 22 a of the walkingtraining apparatus 3 where the leg part of the trainee T is not presentand on which the legs of the assistant A can be placed, and the secondregion R2 where the leg part of the trainee T is included has beenextracted. Thus, it is possible to prevent the foot of the assistant Afrom being erroneously recognized as the foot of the trainee T.Therefore, it is possible to satisfactorily estimate the skeletalcoordinates of the trainee T. The skeleton estimation unit 32 outputsinformation indicating the estimated skeletal coordinates of the traineeT to the abnormality determination unit 33.

On the other hand, if the skeleton acquisition unit 42 of the extractionsystem 5 determine that the skeletal coordinates of the trainee Testimated last time cannot be acquired (NO in S2), it outputsinformation indicating a result of the determination that the skeletalcoordinates of the trainee T estimated last time cannot be acquired tothe skeleton estimation unit 32.

If the skeleton estimation unit 32 receives the information indicatingthat the result of the determination is that the skeletal coordinates ofthe trainee T estimated last time cannot be acquired, it estimates theskeletal coordinates of the trainee T based on the two-dimensional imageand the depth image of the trainee T captured this time, and outputsinformation indicating the estimated skeletal coordinates of the traineeT to the abnormality determination unit 33 (S5). However, as describedabove, when the skeletal coordinates of the trainee T is estimated byusing a two-dimensional image to which a mask has not been applied, itis difficult to estimate the skeletal coordinates of the trainee T insome cases. In such a case, the process proceeds to S6 described later.

Next, the abnormality determination unit 33 determines whether or notthe trainee T is about to be separated from the belt 22 a based on theskeletal coordinates of the trainee T. For example, when the abnormalitydetermination unit 33 recognizes a region where the belt 22 a is presentbased on the two-dimensional image and the depth image and finds thatthe foot of the trainee T enters outside a preset range of the region,the abnormality determination unit 33 estimates that the trainee T isabout to be separated from the belt 22 a, and determines that thewalking of the trainee T is abnormal.

On the other hand, when the abnormality determination unit 33 finds thatthe foot of the trainee T is present within the preset range of theregion where the belt 22 a is present, the abnormality determinationunit 33 estimates that the trainee T is not about to be separated fromthe belt 22 a, and determines that the walking of the trainee T isnormal. The abnormality determination unit 33 may output, for example,information indicating these results of the determination to the controlunit 31.

After that, the abnormality determination unit 33 determines whether ornot to end the abnormal walking determination of the trainee T (S6). Ifthe abnormality determination unit 33 determines to end the abnormalwalking determination of the trainee T (YES in S6), it outputsinformation indicating that the abnormal walking determination is to beended to the control unit 31. When the control unit 31 receives theinformation indicating that the abnormal walking determination is to beended, the control unit 31 controls the motor 14 of the walkingassistance apparatus 2 and the motors 22 b, 25 a, and 26 a of thewalking training apparatus 3 so as to stop the walking training.

On the other hand, if the abnormality determination unit 33 determinesto continue the abnormal walking determination of the trainee T (NO inS6), it outputs information indicating that the abnormal walkingdetermination is to be continued to the control unit 31. When thecontrol unit 31 receives the information indicating that the abnormalwalking determination is to be continued, the control unit 31 controlsthe motor 14 of the walking assistance apparatus 2 and the motors 22 b,25 a, and 26 a of the walking training apparatus 3 so as to continue thewalking training.

As described above, in the extraction system 5, the walking trainingsystem 1, and the extraction method according to this embodiment,regions of the two-dimensional image on both the right and left sides ofthe belt 22 a on which the feet of the assistant A are placed are set tothe first regions R1 in which the leg part of the trainee T is notpresent, and thereby the set regions are distinguished from the secondregion R2 of the two-dimensional image where the leg part of the traineeT is included, and the second region R2 is extracted.

By the above configuration, when the skeleton estimation unit 32estimates the skeletal coordinates of the trainee T based on thetwo-dimensional image, it is possible to prevent the foot of theassistant A from being erroneously recognized as the foot of the traineeT, and as a result, it is possible to accurately extract the secondregion R2 where the leg part of the trainee T is included. Therefore, itis possible to satisfactorily estimate the skeletal coordinates of thetrainee T.

In addition, in the extraction system 5, the walking training system 1,and the extraction method according to this embodiment, the first regionR1 is changed in real time so that the first region R1 where the legpart of the trainee T is not present is separated from the leg part ofthe trainee T by the predetermined distance D in the two-dimensionalimage. By this configuration, it is possible to change the first regionR1 in real time in accordance with movement of the trainee T, and as aresult, it is possible to accurately extract the second region R2 wherethe leg part of the trainee T is present.

Further, in the extraction system 5, the walking training system 1, andthe extraction method according to this embodiment, a color of the maskapplied to the two-dimensional image is set in accordance with thecolors and the materials of the clothes and the shoes of the trainee Tand the assistant A. Therefore, it is possible to prevent the firstregion R1 from being erroneously recognized as the second region R2where the leg part of the trainee T is present.

Other Embodiments

Although the present disclosure has been described as a hardwareconfiguration in the above-described embodiment, the present disclosureis not limited thereto. In the present disclosure, processing of eachcomponent can also be implemented by causing a Central Processing Unit(CPU) to execute a computer program.

For example, the extraction system 5 according to the above-describedembodiment can include the following hardware configuration. FIG. 8 is adiagram showing an example of the hardware configuration included in theextraction system 5.

An apparatus 51 shown in FIG. 8 includes an interface 52, a processor53, and a memory 54. The extraction system 5 described in the aboveembodiment is implemented by the processor 53 loading and executing aprogram stored in the memory 54. That is, this program is a program forcausing the processor 53 to function as the extraction system 5according to the above-described embodiment.

The above-described program can be stored and provided to a computer (acomputer including an information notification apparatus) using any typeof non-transitory computer readable media. Non-transitory computerreadable media include any type of tangible storage media. Examples ofnon-transitory computer readable media include magnetic storage media(such as floppy disks, magnetic tapes, hard disk drives, etc.), andoptical magnetic storage media (e.g., magneto-optical disks). Thisexamples further include CD-ROM (Read Only Memory), CD-R, and CD-R/W.This examples further include semiconductor memories (such as mask ROM,PROM, EPROM, flash ROM, RAM, etc.). Further, the program may be providedto a computer using any type of transitory computer readable media.Examples of transitory computer readable media include electric signals,optical signals, and electromagnetic waves. Transitory computer readablemedia can provide the program to a computer via a wired communicationline (e.g., electric wires, and optical fibers) or a wirelesscommunication line.

The present disclosure is not limited to the above-described embodimentand may be modified as appropriate without departing from the spirit ofthe present disclosure.

For example, in the above-described embodiment, although the firstregion R1 where the leg part of the trainee T is not present is deformedbased on movement of the trainee T, walking training may be performed ina state in which the first region R1 shown in FIG. 5 is fixed.

For example, information indicating the first region R1 output to themask processing unit 44 last time may be stored, the first region R1indicated by the information may be set in the acquired two-dimensionalimage this time, and the first region R1 may be deformed based on theskeletal coordinates of the trainee T estimated last time. By doing so,it is possible to improve the accuracy of extracting the second regionR2 where the leg part of the trainee T is present.

From the disclosure thus described, it will be obvious that theembodiments of the disclosure may be varied in many ways. Suchvariations are not to be regarded as a departure from the spirit andscope of the disclosure, and all such modifications as would be obviousto one skilled in the art are intended for inclusion within the scope ofthe following claims.

What is claimed is:
 1. An extraction system configured to extract aregion of an image where a leg part of a trainee is included, the imagebeing obtained by capturing the trainee who performs walking training ona belt of a treadmill of a walking training apparatus, the extractionsystem comprising: an image acquisition unit configured to acquire animage obtained by capturing the trainee from a front side or a rear sideof the trainee; an extraction unit configured to set regions of theimage on both left and right sides of the belt of the treadmill toregions of the image where the leg part of the trainee is not presentand thereby distinguish the set regions from the region of the imagewhere the leg part of the trainee is included, and extract the region ofthe image where the leg part of the trainee is included; and a maskprocessing unit configured to apply a mask to each of the regions of theimage where the leg part of the trainee is not present.
 2. Theextraction system according to claim 1, further comprising a skeletonacquisition unit configured to acquire skeletal coordinates of the legpart of the trainee in real time, wherein the extraction unit changesthe regions of the image where the leg part of the trainee is notpresent in real time so that the regions of the image where the leg partof the trainee is not present are separated from the leg part of thetrainee by a predetermined distance based on the skeletal coordinates ofthe leg part of the trainee.
 3. The extraction system according to claim2, wherein the extraction unit changes the regions of the image wherethe leg part of the trainee is not present in real time so that theregions of the image where the leg part of the trainee is not presentare separated from a straight line by the predetermined distance, thestraight line connecting a position of a hip joint of the trainee to aposition of a knee joint of the trainee.
 4. The extraction systemaccording to claim 1, wherein the mask processing unit sets a color ofthe mask based on clothes or shoes worn by the trainee.
 5. A walkingtraining system comprising: the extraction system according to claim 1;and a walking training apparatus comprising a treadmill for the traineeto perform walking training, wherein the walking training apparatusfurther comprises: a skeleton estimation unit configured to estimate theskeletal coordinates of the leg part of the trainee based on the imageto which the mask has been applied; and an abnormality determinationunit configured to determine whether or not an abnormality of thetrainee has occurred based on the skeletal coordinates of the leg partof the trainee.
 6. An extraction method for extracting a region of animage where a leg part of a trainee is included, the image beingobtained by capturing the trainee who performs walking training on abelt of a treadmill of a walking training apparatus, the extractionmethod comprising: acquiring an image obtained by capturing the traineefrom a front side or a rear side of the trainee; setting regions of theimage on both left and right sides of the belt of the treadmill toregions of the image where the leg part of the trainee is not present,thereby distinguishing the set regions from the region of the imagewhere the leg part of the trainee is included, and extracting the regionof the image where the leg part of the trainee is included; and applyinga mask to each of the regions of the image where the leg part of thetrainee is not present.
 7. The extraction method according to claim 6,further comprising estimating skeletal coordinates of the leg part ofthe trainee in real time, wherein the regions of the image where the legpart of the trainee is not present are changed in real time so that theregions of the image where the leg part of the trainee is not presentare separated from the leg part of the trainee by a predetermineddistance based on the skeletal coordinates of the leg part of thetrainee.
 8. The extraction method according to claim 7, wherein theregions of the image where the leg part of the trainee is not presentare changed in real time so that the regions of the image where the legpart of the trainee is not present are separated from a straight line bythe predetermined distance, the straight line connecting a position of ahip joint of the trainee to a position of a knee joint of the trainee.9. The extraction method according to claim 6, wherein a color of themask is set based on clothes or shoes worn by the trainee.
 10. Anon-transitory computer readable medium storing an extraction programfor extracting a region of an image where a leg part of a trainee isincluded, the image being obtained by capturing the trainee who performswalking training on a belt of a treadmill of a walking trainingapparatus, the extraction program causing a computer to: acquire animage obtained by capturing the trainee from a front side or a rear sideof the trainee; set regions of the image on both left and right sides ofthe belt of the treadmill to regions of the image where the leg part ofthe trainee is not present and thereby distinguish the set regions fromthe region of the image where the leg part of the trainee is included,and extract the region of the image where the leg part of the trainee isincluded; and apply a mask to each of the regions of the image where theleg part of the trainee is not present.